Skip to content

feat: support concurrent chunk uploads#116

Open
TorstenDittmann wants to merge 13 commits into
mainfrom
concurrent-chunk-uploads-1-9-x-minimal
Open

feat: support concurrent chunk uploads#116
TorstenDittmann wants to merge 13 commits into
mainfrom
concurrent-chunk-uploads-1-9-x-minimal

Conversation

@TorstenDittmann
Copy link
Copy Markdown
Contributor

This PR updates the SDK to support concurrent chunk uploads.

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented May 21, 2026

Greptile Summary

This PR refactors the chunked file upload logic in both client_browser.dart and client_io.dart to upload up to 8 chunks concurrently using Future.wait, while always uploading the first chunk sequentially to establish the uploadId. The version is bumped from 24.1.0 to 24.2.0 across pubspec.yaml and README.md.

  • The worker pool uses a shared nextChunk counter with a while/++ dispatch pattern; this is safe in Dart's single-threaded event loop since the check-and-increment is synchronous between await points.
  • In client_io.dart, each concurrent worker opens its own RandomAccessFile (rather than re-opening per-chunk as before), which is a meaningful improvement for file-based uploads.
  • Three issues flagged in prior review threads remain unresolved: uncancelled in-flight workers when one chunk fails, a finalResponse/lastResponse tracking path that can return stale data, and the per-open-close pattern for the initial sequential chunk in client_io.dart.

Confidence Score: 3/5

The concurrent upload logic introduces real correctness risks that have been flagged but not yet addressed; merging as-is could silently corrupt a resumed upload after a transient failure.

Three substantive defects identified in prior review rounds are still present: (1) when any uploadChunk call throws, Future.wait surfaces the error but remaining workers keep running, mutating shared state and potentially landing chunks on the server so a retry resumes from the wrong offset; (2) finalResponse is only set inside uploadNext, so if isUploadComplete never fires the caller receives the first chunk response instead of the completed upload metadata; (3) the first sequential chunk in client_io.dart still opens and closes its own RandomAccessFile independently.

lib/src/client_browser.dart and lib/src/client_io.dart - specifically the uploadNext worker loop and Future.wait call in both files.

Important Files Changed

Filename Overview
lib/src/client_browser.dart Rewrites chunked upload to use up to 8 concurrent workers via Future.wait; shared nextChunk/completedChunks/uploadedBytes counters are safe under Dart's single-threaded event loop, but previously-flagged issues (uncancelled workers on failure, stale lastResponse/finalResponse) remain unresolved.
lib/src/client_io.dart Same concurrent upload refactor as client_browser.dart; each worker opens its own RandomAccessFile (improvement over old per-chunk open/close), but the first sequential chunk still opens its own handle, and the same uncancelled-worker-on-failure issue applies.
pubspec.yaml Version bump from 24.1.0 to 24.2.0 - no logic changes.
README.md Updated dependency version reference from 24.1.0 to 24.2.0 - documentation only.

Reviews (6): Last reviewed commit: "Commit from GitHub Actions (Format and p..." | Re-trigger Greptile

Comment thread lib/src/client_browser.dart
Comment thread lib/src/client_io.dart Outdated
}

final concurrency = min(8, chunks.length);
await Future.wait(List.generate(concurrency, (_) => uploadNext()));
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Uncancelled workers continue uploading after a failure

Future.wait with the default eagerError: true propagates the first error immediately, but does not cancel the remaining uploadNext futures — they keep running on the event loop. Concretely: if chunk N receives a 5xx from the server, the caller's await throws, but up to 7 other workers are still sending HTTP requests. onProgress keeps firing on the closed-over closures, and completedChunks/lastResponse keep mutating. If the caller catches the error and retries, the resume-offset query will see a higher chunksUploaded than anticipated (those background requests may have been accepted by the server), leading to duplicate-chunk or skipped-chunk scenarios. The sequential implementation avoided this entirely because a thrown exception exited the loop immediately. The same issue is present in client_io.dart at the equivalent line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants